Programmable multiple-point illuminator, confocal filter, confocal microscope and method to operate said confocal microscope

ABSTRACT

A programmable multiple-point illuminator for an optical microscope (M) including a light source and a spatial light modulator (SLM) to modulate a light beam from the light source. The modulated light beam scans across a sample placed under the microscope objective, the sample being provided with fluorophores. The SLM includes a first acousto-optic deflector and a second acousto-optic deflector, the first acousto-optic deflector having a first modulation plane and the second acousto-optic deflector having a second modulation plane, said two acousto-optic deflectors being arranged in cascade to provide respective deflection in different directions, whereby the SLM is enabled to scan in two dimensions across the sample. The SLM further a telescope relay to conjugate the first modulation plane with the second modulation plane. The illuminator also has an arbitrary waveform generator that is configured to synthesize holograms, and is arranged to simultaneously inject a first such hologram into the first acousto-optic deflector and a second such hologram into the second acousto-optic deflector, in order for the SLM to modulate the light beam in response to said holograms.

TECHNICAL FIELD

The present disclosure is related to a programmable multiple-pointilluminator for an optical microscope, and to a confocal filter to makea microscope a confocal one. The disclosure is also related to a methodto operate a confocal microscope.

The illuminator includes a light source and a spatial light modulatorfor modulating a light beam from the light source, the modulated lightbeam being provided for scanning across a sample placed under anobjective of the microscope, the sample being normally provided withfluorophores. The expression ‘the sample is placed under the microscopeobjective’ means that the light beam is to be focused by the objectivein the sample (that expression is not to be understood as meaning thatthe sample is always located below the objective).

BACKGROUND

Confocal microscopy is the reference technique for sample visualizationin all fields of cellular biology, and it is widely acknowledged as oneof the most important inventions ever made in optical microscopy.Confocal microscopes have enjoyed a tremendous explosion in popularityin recent years and most universities and scientific institutionsworldwide, and increasingly many individual laboratories, own confocalmicroscopes.

Confocal microscopes come essentially in two different modalities:single-point and multi-point scanning instruments. They are often usedin combination with fluorescent tags (fluorescent molecules orfluorophores) that selectively label the structure of interest and thatrespond to an illumination laser by emitting light at a longerwavelength (Stokes shift). This wavelength shift permits an easyisolation of the excitation and emission optical trains by dichroicmirrors and filters.

Single-point confocal microscopes are based on a single laser beam thatprogressively scans the sample on a point-by-point basis, which resultsin a high resolution, high contrast and optically-sectioned image afterthe light emitted by the sample is filtered out by a small pinholeaperture conjugated to the laser spot. Light emitted by excitedfluorophores above and below the focused plane are intercepted by thepinhole and do not reach the detector, minimizing the light haze thatplagues non-confocal microscopes when imaging thick samples.

However, this point-by-point scanning method entails an obviously slowimage acquisition, which is the main limitation of single-pointconfocals. Live samples such as cells have often to be fixed (i.e.killed) to obtain images without motion blur, as the instrument isunable to resolve the temporal dynamics of many cellular phenomena.

Faster scanning has been therefore a crucial vector in the developmentof modern confocal microscopy. However, scanning using a single laserspot cannot be made arbitrarily fast: a high scan rate means that thelaser spot can only illuminate any sample point during a very shortexposure time, in the scale of microseconds. In order to compensate forthis small excitation time, the laser power falling on the sample has tobe increased, which very quickly saturates the fluorophores. Laser powerincrements above the saturation threshold do not result in an equivalentincrement in the fluorescent emission rate, in such a way that the totalamount of photons reaching the detector will decrease with decreasingexposure times, thus setting a limit to the scanning speed around a fewframes per second.

The only technical solution that enables a fast confocal operation isthe use of several laser spots scanning the sample in parallel.

Multi-point confocals have been developed in response to this need. Theyuse thousands of laser beamlets to simultaneously scan the sample, thusbeing able to reach frame rates in the range of hundreds of frames persecond. An added advantage of splitting the total power into many laserfoci is that these instruments are considerably gentler with thebiological samples ( 1/15 less damaging than a single point confocal incomparable conditions), minimizing photobleaching and phototoxicity.

However, commercial implementations of the multi-point scanningprinciple are based on disks covered by arrays of tiny microlenses andpinholes (Nipkow disk) that spin at high speed, which make the systeminflexible and optically inefficient. Indeed, spinning disk microscopescannot scan arbitrary regions of interest in the sample and are matchedto a single objective, usually with high magnification and highnumerical aperture lenses. Also, a typical Nipkow disk has around 4%optical efficiency, requiring powerful excitation lasers, which arecostly.

A further difficulty is the reduced confocality arising from thecrosstalk between pinholes, especially in thick samples (crosstalk isdue to leakage on a detected signal from other optical signals).Spurious light excited by one laser spot can reach the detector (e.g. acamera) through neighbouring pinholes, resulting in a noticeably lowerresolution image when compared with that produced by a single-pointconfocal in the same conditions.

Aims have been attempted to suppress optical crosstalk betweenillumination spots produced to inspect a sample. E.g., a diffractiveoptical element (DOE) has been disclosed positioned before an objectivelens. The DOE makes copies of the spot output without changing the spotspacing, in order to ensure sufficient separation between spots. Butprevious attempts only envisage simple and relatively inflexible spotpatterns (e.g., one programmable acousto-optic deflector just to providean adjustable spot spacing), thus reproducing the drawbacks of theNipkow disk to some extent.

In sum, the two modalities of confocal microscopy have clear advantagesand disadvantages with respect to each other, which make themspecialized and not interchangeable. In general, users need access tothe two types of instrument at one time or another.

SUMMARY

An aspect of the present disclosure may include bridging the gap betweensingle-point and multi-point microscopy, making it possible forlaboratories to envisage unlimited operation with just one confocalmicroscope.

Another aspect of the present disclosure may include transcendingconventional multi-point microscopy by creating free illuminationpatterns that are significantly more complex than mere spot arrays.

In a further aspect, a programmable multiple-point illuminator for anoptical microscope may include a light source and a spatial lightmodulator (SLM) to modulate a light beam from the light source. Themodulated light beam scans across a sample placed under the microscopeobjective, the sample being provided with fluorophores. The SLM mayinclude a first acousto-optic deflector (AOD) and a second acousto-opticdeflector, the first AOD having a first modulation plane and the secondAOD having a second modulation plane, said two acousto-optic deflectorsbeing arranged in cascade to provide respective modulation (i.e.deflection) in different directions (e.g. the respective directions ofdeflection of the two AODs may be orthogonal), whereby the spatial lightmodulator is enabled to scan in two dimensions across the sample. TheSLM may further include a telescope relay to conjugate the firstmodulation plane with the second modulation plane. The illuminator mayalso include an arbitrary waveform generator (AWG) that is configured tosynthesize radiofrequency (RF) signals computed with digital holographyalgorithms, said synthesized signals being termed holograms (such ahologram contains a coded record of an optical wave, including itsamplitude and phase properties), and is arranged to simultaneouslyinject a first such hologram into the first AOD and a second suchhologram into the second AOD, in order for the SLM to modulate the lightbeam in response to said holograms.

The illuminator can thus illuminate the sample with a 2D light patterndesigned and chosen with a precise purpose, and so can bridge the gapbetween single-point and multi-point microscopy because the selectedlight pattern can be very simple (e.g. with a few spots) or very complex(e.g. with a dense and complicated spot arrangement), and can beanalogously generated in either case.

In an example, the programmable illuminator may include a laser device(light source) that can dynamically project onto the microscopic samplea plurality of light spots in parallel with accurate positioning,according to an arbitrarily capricious and discretionary pattern. Theprogrammable illuminator is based on AOD technology. AODs areessentially ultrafast light deflectors that can impart a change in thedirection of a light beam that crosses the device. An AOD includes apurposely cut optical crystal and a piezoelectric transducer that isattached at one end of the crystal and can create sound waves therein.

The modulation plane (or pivot plane) of an AOD is an imaginary planeinside the AOD crystal at which an incoming collimated light beamappears to be deflected, resulting in an outgoing collimated light beamtravelling at a different angle. The modulation plane can be found byforward-projecting the propagation direction of the incoming light beamand back-projecting the propagation direction of corresponding outgoinglight beam; the back and forward beams meet in a plane (the modulationplane) inside the crystal.

The two optically conjugated AODs produce a joint modulation functionthat is separable, i.e., it is the product of the modulation function ofthe first AOD (e.g. in the X direction) and the modulation function ofthe second AOD (e.g. in the Y direction).

It is known to apply a simple, sinusoidal radiofrequency (RF) signal(see FIG. 2A) to the piezoelectric transducer. The transducer vibrationlaunches a sound signal into the crystal that spatially modulates therefractive index in a periodic way, creating a diffraction grating. Byvarying the frequency of the control RF signal, the period of thediffraction grating is modified and the deflection of the laser can bequickly changed in such a way that the beam can be redirected to adifferent spatial location.

In an example, the AWG is configured to synthesize holograms of anarbitrary complexity (see FIGS. 2B, 3 and 4), intended to enable theilluminator to scan across the sample with an arbitrarily complextwo-dimensional light pattern. AWGs are able to synthesize electricalsignals with arbitrary shapes, that is, in such a way that the temporalvariation of the output voltage can be specified by the user in acompletely general manner within the bandwidth of the instrument. Inthis sense, AWGs are a generalization of voltage-controlled oscillators(VCOs), which can only produce sinusoidal functions of variablefrequency, and function generators, which produce several differentwaveforms (sine, square, sawtooth, etc.) but only within the optionspre-defined by design.

In general, AWGs are composed of a digital device capable ofmathematically synthetizing a waveform, such as a Field-programmableGate Array (FPGA), and a fast digital-analog converter circuitry thateventually produces the time-varying electric signal.

In an example, the Illuminator may include a scanning lens arrangedafter the spatial light modulator to project a reconstruction of adesired illumination pattern onto an intermediate image plane, thescanning lens forming, together with a tube lens of the microscope, a 4foptical system to conjugate the modulation planes of the acousto-opticdeflectors with the input pupil of the microscope objective, which is aFourier transform lens, said objective being charged with focusing thelight beam on a Fourier reconstruction plane that intersects the sample,so that the centring of the hologram is unimportant due to the shiftingproperty of the Fourier transform.

In a further aspect, a confocal filter for an optical microscope havingsuch an illuminator includes an imaging sensor provided with anelectronic multi-pixel detector configured to enable real-timeimplementation of one digital pinhole around the image of any excitedfluorescence location (by virtue of the fluorophores) in the sample. Thefilter further includes a relay system to focus the fluorescent lightemitted by the sample on the imaging sensor.

The confocal filter, together with the programmable illuminator, makesthe optical microscope actually confocal, as a digital post-processingof the image can mimic the effect of physical pinholes. For example, aCMOS image sensor, in which programmable pixel sets (e.g. rows) can beselectively read, may make image filtering fast enough.

In an example, the set of the programmable illuminator and the confocalfilter may include a device to synchronize the AWG with the imagingsensor in order to correctly compose a confocal emission image.

In a still further aspect, a method to operate a confocal microscopeincluding the programmable illuminator and the confocal filter includes:

-   -   making the light source to emit a first light beam of a certain        diameter;    -   expanding the first light beam into a second light beam having a        prescribed diameter to define an illumination window on the        first AOD;    -   injecting a first hologram into the first AOD, in order to        modulate the second light beam and transform it into a third        light beam;    -   imaging the third light beam on the second modulation plane;    -   collimating the third light beam at zero modulation, so that the        diameter of the third light beam is that of the second light        beam and defines an illumination window on the second AOD;    -   injecting a second hologram into the second AOD, in order to        modulate the third light beam and transform it into a fourth        light beam;    -   imaging the fourth light beam on the input pupil of the        microscope objective;    -   focusing the fourth light beam on a reconstruction plane that        intersects the sample;    -   collecting the fluorescent light emitted by the sample and        focusing said light on the imaging sensor.

The method may further include:

-   -   computing the first synthetic radiofrequency signal with a first        digital holography algorithm and the second synthetic        radiofrequency signal with a second digital holography algorithm        (although the first and second digital holography algorithms may        be the same algorithm);    -   synthetizing, by the AWG, the first hologram, to be injected        into the first AOD, from said first computed signal, and the        second hologram, to be injected into the second AOD, from said        second computed signal.

In operation, the AWG injects into the AOD cells mathematicallydesigned, synthetic RF signals, so that the laser beam becomesadditionally modified. This allows the creation of desired, arbitrarilycomplex light patterns in contrast to the simple deflections ofconventional AODs. For example, the laser beam can be split into severalsub-beams and their spatial locations can be individually controlled.This principle can be used to parallelize the illumination of a confocalmicroscope, similarly to what the spinning disk achieves with theperforated Nipkow disks.

The sample is scanned by shifting the array of illuminating spots untilit has been fully exposed, which only requires sending new controlsignals to the AODs. In contrast to the spinning disk confocalmicroscope, the sampling patterns are herein totally programmable, asthey are not based on fixed apertures etched on a solid substrate inmechanical motion.

A difficulty with this approach is the confocal filtering ofout-of-focus light that cannot be done with the AODs themselves. A wayto overcome this is by creating virtual pinholes, as explained above(and below, in the detailed description).

Regarding applications where the sample needs to be illuminated by anon-separable light pattern, there are disclosed herein two proceduresto compose a general light distribution as a sum of severalmathematically separable patterns. An important aspect to consider isthat the patterns are time-multiplexed, that is, they are generated, andthus summed, by an integrating device that is sensitive to irradiance(e.g. a camera, or the sample itself), at different times. Consequently,the patterns to be added can only have positive values; subtraction canbe implemented through optical devices but that requires a coherentsuperposition that does not take place here. The disclosed algorithmsare:

a) Decomposition into Lines

An instant way to decompose a two-dimensional N×M image into separablepatterns is to divide it into lines (i.e. its N rows or M columns). Forthe sake of clarity, let us assume that we divide the image into its Nrows. The image can be composed line by line if the AOD that deflects inthe X direction reconstructs the line intensity and, at the same time,the AOD that modulates in the Y direction deflects that line to itsappropriate Y position (these two AODs are orthogonal).

Then the driving signal of the Y-direction AOD is a continuoussinusoidal pattern, whose frequency changes incrementally and definesthe reconstruction position of the line. The X-direction AOD is thehologram encoding the inverse Fourier transform of the particular linethat is being reconstructed at that moment.

b) Decomposition into Separable Two-Dimensional Sub-Images: TheNon-Negative Singular Value Decomposition (NNSVD)

The singular value decomposition (SVD) is a well-known factorizationmethod that decomposes an N×M matrix A into a sum of outer products,i.e. A=Σ₁ ^(k) ω_(i) u_(i) v_(i) ^(T), where k is the rank of A.Considering an image as a matrix, the SVD operation automaticallyproduces a decomposition into separable patterns (the u and v vectors ofthe outer product). However, these patterns contain in general positiveas well as negative values, which cannot be implemented through the AODprojection.

Fortunately, a non-negative singular value decomposition (NNSVD) can bedefined and computed through an iterative algorithm as disclosed in W.Liu et al., “Nonnegative Singular Value Decomposition for MicroarrayData Analysis of Spermatogenesis”, Proceedings of the 5th InternationalConference on Information Technology and Application in Biomedicine,225-228 (2008).

It has been found that the NNSVD algorithm is very useful to decomposeimages into separable sub-images to be displayed by the AOD illuminator,while compressing the image information what allows us to increase thereconstruction speed in certain cases. The algorithm approximates thediagonalization of an image A with throughout positive values, accordingto: A≈X·W·Y

The diagonal matrix W includes the eigenvalues of the diagonalization.The factor w weights the outer product of the one-dimensional vectorsx_(i) and y_(i). When the columns x_(i) and y_(i) are normalized, thenw_(ii) represents the total power (or energy) of the decompositionsub-images. This property is useful when determining the significance ofa sub-image. Sub-images with a negligible intensity may be discarded toaccelerate the reconstruction and to improve the efficiency.

BRIEF DESCRIPTION OF DRAWINGS

Non-limiting examples of the present disclosure will be described in thefollowing, with reference to the appended drawings, in which:

FIG. 1 shows a setup for an optical microscope M;

FIG. 2 shows a comparison between two radiofrequency signals;

FIG. 3 represents a 2D acousto-optic deflector;

FIG. 4 shows a comparison between two holograms;

FIG. 5 shows an array of light spots;

FIG. 6 illustrates a confocal filtering;

FIG. 7 shows a comparison between three imaging results;

FIG. 8 shows a comparison between six image reconstructions;

FIG. 9 shows a hologram transition; and

FIG. 10 shows a comparison between different phase alignments.

DESCRIPTION

FIG. 1 shows an illumination system including a first laser source 1 anda second laser source 2, a dichroic device 4 to couple respective beamsfrom the laser sources into a single direction, a first acousto-opticdeflector (AOD) 8, an inverted telescope 5 before the first AOD 8, asecond AOD 9, a telescope relay 10 between the first AOD 8 and thesecond AOD 9, and a scanning lens 17 after the second AOD 9.

FIG. 1 also shows the following parts of an optical microscope M: a tubelens 19 and an input pupil 20 to an objective lens 21 (the microscopeobjective).

The illumination system of FIG. 1 further includes an arbitrary wavegenerator (AWG) 13, a first radiofrequency (RF) amplifier 14 in the wayfrom the AWG 13 to the first AOD 8, a second radiofrequency (RF)amplifier 15 in the way from the AWG 13 to the second AOD 9, atwo-dimensional (2-D) imaging sensor 26, a synchronizer 16 between theimaging sensor 26 and the AWG 13, a relay system 25 before the imagingsensor 26, a fluorescence filter 24 before the relay system 25, and adichroic mirror 23 between the tube lens 19 and the fluorescence filter24 (the dichroic mirror 23 is also located between the scanning lens 17and the tube lens 19).

Any laser source 1 or 2 can be either continuous or pulsed. Pulsedlasers (femtosecond) can be used to cause a multi-photon absorptionphenomenon on a reconstruction plane with a view to excite fluorescenceor induce photo-polymerization in some applications. When a pulsed laseris used as the illumination source, additional optical elements, such asa prism 3, can be used to avoid or compensate large group velocitydispersion inside the AODs. In contrast, continuous-wave (CW) lasersemit one, uninterrupted light beam, and are preferable for generalfluorescence microscopy as they have much lower peak-power values thanpulsed or ultrafast lasers, which may be detrimental to the samples, andpresent a simpler design that make them easier to manufacture andmaintain.

In general, several laser sources can provide illumination at selectedwavelengths for polychromatic applications, either simultaneously orsequentially, and can be coupled into a single direction via thedichroic device 4. For example, several light sources are necessary forexcitation in multi-colour microscopy, in which at least two fluorescentmolecular dyes are used to label different sample structures. Upon laserirradiation, these structures emit light within distinct wavelengthranges so that they can be individually visualized. Moreover, two laserswith different wavelengths may be necessary for the excitation anddepletion stages in super-resolution techniques such as STED (stimulatedemission-depletion) or RESOLFT (reversible saturable opticalfluorescence transitions) microscopies. Besides, multiple lasers can beused to study the co-localization, within the same biological structure,of two or more molecular species, which are made visible by labellingdyes that respond to the different excitation wavelengths.

In operation, the single or combined laser beam is expanded by theinverted telescope 5 from an initial laser diameter D₁ to anillumination window of diameter D₂ on the first AOD 8. The size D₂ ofsaid illumination window, and therefore the magnification of the beamexpander 5, is carefully selected as this controls the field of viewversus the frame rate trade-off of the illumination system.

The two AODs 8 and 9 provide modulation of the light beam in twoorthogonal directions (X and Y), that is, they constitute a spatiallight modulator. These AODs are high-resolution, high-deflection angledevices (preferably providing higher than 500×500 resolvable spots),with a large square input window of preferably more than 8×8 mm, andwith as similar as possible acousto-optic properties. When illuminatedby several lasers, the AODs are configured to work in the Bragg regimesimultaneously for the whole set of wavelengths involved.

Furthermore, when addressed by specific sets of RF signals within thebandwidth of the AOD device, the two AODs must provide an overlappingdeflection range for the whole set of wavelengths. The AODs are mountedon tip-tilt opto-mechanical mounts (not shown) to be oriented inappropriate angles to the incoming laser beam, in order to achieve gooddiffraction efficiency for the whole bandwidth and for any wavelengthinvolved.

The two orthogonal AODs are optically conjugated by the telescope relay10, which include two identical lenses in a 4f configuration to imagethe modulation plane 81 of the first AOD 8 (i.e. the pivot plane of thebeam deflection) into the modulation plane 91 of the second AOD 9 withunit magnification. Furthermore, the relay 10 simultaneously keeps thelaser beam collimated (at zero modulation) with diameter D₂ at anillumination window on the second AOD 9. The two optically conjugatedAODs then achieve an optical multiplication of their modulationfunctions as follows: h(x,y)=f(x)·g(y) (see FIG. 3), without anydistortion caused by light propagating from the first AOD 8 to thesecond AOD 9.

The AOD devices are connected to a dual-channel arbitrary waveformgenerator (AWG) 13 through radio-frequency (RF) amplifiers 14 and 15.The AWG digitally synthetizes two, usually discrete, RF signals computedwith techniques of digital holography. These synthetic pixelated radiosignals (see FIG. 3A) induce a complex spatial variation of therefractive index inside the AOD cells, by the acousto-optic effect,which in turn modulate the light beam in a way akin to a Fourier spatiallight modulator. As a result, the complex refractive index modulationbecomes an acoustically induced hologram.

The AWG 13 can produce signals with a bandwidth matching that of the AODdevices (8, 9) and incorporates a memory bank capable of storing enoughpre-computed driving signals to jointly represent complex lightpatterns. In effect, in contrast to a true two-dimensional spatial lightmodulator, the AOD device described herein includes two one-dimensionallight modulation devices in cascade, which can only producetwo-dimensional light patterns that are mathematically separable (i.e.,which are the product of a function in X by a function in Y,h(x,y)=f(x)·g(y), see FIG. 3). This is usually enough to implementexcitation patterns including light spot arrays that can scan the samplein parallel (as explained above, the sample is placed under theobjective 21 of microscope M).

When the illuminator is needed to project more complex light patterns(to excite an arbitrary region of interest, for opto-stimulation orphotobleaching, for example), these can be obtained by seriesdecomposition into separable functions through an appropriatemathematical algorithm (represented by ref. 11 in FIG. 1), i.e.:H(x,y)=Σ_(i)h_(i)(x,y)=Σ_(i)f_(i)(x)·g_(i)(y).

The sample itself, through a cumulative effect, will add up the seriesterms (or the final detector in other applications, usually throughsynchronization (ref. 16) with the AWG 13).

In either case, in order to correctly form the desired light pattern,one must consider that the optical modulation within the AOD devices isproduced by travelling sound waves, which propagate from thepiezoelectric transducer at one end of the acousto-optic crystal to asound absorber at the other end, interacting with the laser beam withina finite time interval when crossing the illuminated optical window ofthe AOD. This necessarily requires that the reconstruction be carriedout at a Fourier plane with respect to the conjugated modulation planes(81 and 91) of the two AODs 8 and 9, so that the centring of thehologram is unimportant due to the shifting property of the FourierTransform. In order to do this, the modulation planes need to beconjugated with the input pupil 20 of the microscope objective 21, thatwill act here as a Fourier transform lens.

First of all, the scanning lens 17 projects a reconstruction of thedesired excitation pattern into an intermediate image plane 18. Thisscanning lens 17, together with a tube lens 19 of the microscope M, forma 4-f system that optically conjugates the modulation planes 81 and 91(which are in turn mutually conjugated by the telescope relay 10) withthe input pupil 20 of the microscope objective 21, which focuses thelaser beam on a final reconstruction plane 22 that intersects the sample(not shown), thereby exciting the fluorescence of the sample (i.e. thefluorophores therein).

The field of view of the microscope objective 21 (which is a Fouriertransform lens) must be larger than the inclinations of the highestspatial frequency Fourier components contained in the displayedholograms. Pupil matching may be necessary in order to avoid vignettingof said Fourier components inside the microscope objective 21. Since themicroscope objective 21 must be, in general, a highly corrected opticalsystem composed of several optical surfaces, its input pupil 20 may lieinside the system and not be directly accessible. Since the modulationplanes 81 and 91 also lie inside the AOD devices 8 and 9, respectively,the relay system formed by the scanning lens 17 and the tube lens 19 isused to couple the modulations planes to the input pupil 20 of themicroscope objective 21.

Additionally, the scanning lens 17 plus the tube lens 19 match the inputpupil size D₃ with the AODs optical window size D₂, in order to use thefull numerical aperture of the microscope objective 21 and therebyoptimizing the sectioning capability and resolution. The ratio D₃/D₂then determines the magnification of the telescopic system formed by thescanning and tube lenses 17 and 19, thus the deflection angles of theFourier components of the wavefronts diffracted by the AODs and,consequently, the field of view of the illuminator on the sample plane.As the time T that the sound wave needs to cross the illuminated windowis T=D₂/v, where v is the speed of sound in the crystal, D₂ is alsorelated to the maximum repetition rate at which holograms can be updated(i.e. the maximum frame rate of the illuminator), so that it should becarefully selected.

The fluorescent light emitted by the sample, travelling in the oppositedirection to the excitation laser, is collected by the microscopeobjective 21 and focused onto the intermediate image plane 18. Afterthis, the dichroic mirror 23 and the fluorescence filter 24 select theemission wavelength, and the relay system 25 focuses the emission lighton the 2D imaging sensor 26. Electronic post-processing of severalindividual frames obtained by shifting the multi-spot illumination arrayin an orderly fashion (see FIG. 5) can produce a final confocal image asillustrated in FIGS. 6 and 7. A precise synchronization (16) between thesensor 26 and the AWG 13 is necessary for correctly composing the finalimage.

The imaging sensor may be a CMOS multi-pixel detector that allowsarbitrary reading regions at high speeds enabling the real-timeimplementation of digital masks (digital pinholes) around each emissionfocus. Different algorithms that improve both lateral and axialresolution, such as photon reassignment, can also be implemented.

FIGS. 3A and 3B illustrate the formation of a two-dimensional modulationpattern out of two one-dimensional modulation signals injected intoorthogonal AOD cells. As the two AOD devices are conjugated through therelay 10, the first, X-varying, modulation plane is projected into thesecond, Y-varying, modulation plane. The result is an outer vectormultiplication of the discrete signals encoded in the two planes.

FIG. 4 illustrates the last steps in the synthesis process of theholographic radiofrequency signals that are injected into the AOD cells.Firstly (and not shown in FIG. 4), the illumination patterns aredescribed with digital data arrays. The index of each array elementrepresents the position of the physical illumination spot in the sampleplane, and the value of each array element represents its intensityaccordingly. If the illumination pattern I(x,y) is separable, it can bedescribed by the multiplication of two discrete functionsI(x,y)=I₁(x)·I₂(y). If this is not the case, a decomposition followingthe procedures disclosed above may be carried out.

As the laser beam wavefront is modulated by the AODs and thentransformed into the desired illumination pattern by an optical Fouriertransform, the calculation of the required wavefront modulation can beperformed via an inverse Fourier transform of the illumination pattern.Since the illumination pattern is described digitally, the requiredamplitude and phase modulation may be calculated by the discrete Fouriertransform (DFT). The position of the illumination pattern elements(spots) is controlled by the frequencies at which the DFT is evaluated.

In order to obtain the required spatial amplitude and phase modulationby the AODs, an electronic driving signal is required that arouses thecorresponding acousto-optic modulation in the AOD crystal. Finally, therelationship between the driving RF signal and the resulting spatialamplitude and phase modulation is simple: a piecewise defined sinusoidaldriving signal with a carrier frequency f_(c), and piecewise varyingamplitude and phase corresponding to the calculated amplitude and phaseresults (to a good approximation) in the desired wavefront modulation(ref. A in FIG. 4).

Additionally, an iterative Gerchberg-Saxton algorithm (ref. B in FIG. 4)is preferably used to convert the full-complex hologram (left graph inFIG. 4) into a kinoform (or phase-mostly hologram, right graph in FIG.4) in order to maximize the optical efficiency. The typical result ofthe RF frequency synthetized by this procedure is shown in the rightgraph of FIG. 4, which represents a high-efficiency hologram (the leftgraph of FIG. 4 represents a relatively low-efficiency hologram) that isinjected into one AOD 8 or 9 (ref. C in FIG. 4).

FIG. 2A shows an oscilloscope trace of a pure sinusoidal RF signal of afrequency=75 MHz that typically drives AODs when used as a beamdeflector. FIG. 2B, by contrast, shows a much more complex signalcomputed with the techniques of digital holography explained above,designed to produce the arbitrary light distributions disclosed herein.

By these holographic RF signals, arbitrary light distribution can beobtained on the reconstruction plane 22. When the illuminator of thepresent disclosure is used to excite fluorescence from a microscopicspecimen, a convenient pattern is formed by a regular array of lightspots (for example, a square matrix of 32×32 light spots). The array canbe incrementally shifted, by changing the X and Y holograms, until thesample becomes fully exposed, as illustrated in FIG. 5. The whole sampleplane can be scanned by an illumination pattern of a regular array oflight spots that is incrementally shifted horizontally and verticallyuntil the gaps between spots are filled. The shift increment can beselected as the radius of the individual light spots in agreement withRayleigh criterion.

In other words, the increment in the X and Y directions can be selectedaccording to the resolution of the optical system, i.e. matching theradius of the point-spread function at the reconstruction plane 22 (forexample, 16×16 shifts). The specimen (in the sample) emits fluorescencein response to the individual excitation spot arrays, which is capturedby the sensor 26 in synchrony (16) with the AWG 13.

FIG. 6 illustrates one of the emitted fluorescence images in response tothe illumination light. The magnified inset shows that the resultingimage is also an array of light spots, whose intensity is related to thelocal density of fluorescence molecules at the focal plane and which issurrounded by a cloud of scattered light mostly coming from out-of-focusplanes. This scattered light, if not eliminated, greatly reduces thecontrast and resolution of the final image, as illustrated in FIG. 7.Confocal filtering is carried out in the single emission frames.Fluorescent photons coming from planes above and below the plane ofinterest produce halos of light around the excitation foci, which can beeliminated by digitally processing the images: only the informationinside a small circle around each light spot is kept, the rest of theimage is erased.

FIG. 7 shows again an individual response of the specimen to one of theexcitation spot arrays. There are shown fluorescent images of the actinnetwork in a chicken embryo immunolabeled with phalloidin+TRITC. Fromleft to right: single emission frame (the excitation pattern being ansquare array of 32×32 laser spots, FIG. 7A), widefield (non-confocal)image (FIG. 7B), confocal image obtained after pinhole filtering andaddition of the whole set of single emission frames (FIG. 7C).

If the final image is composed as the addition of these individualresponses, without filtering the out-of-focus light, the image in FIG.7B results. By contrast, if the individual fluorescence frames aredigitally pinholed, by keeping only the light within a small circlearound the excitation foci (see the rightmost image in FIG. 6) beforethe final composition, FIG. 7C results, which clearly shows improvedresolution and contrast. The spot array illumination permits thisoperation to be carried out, which justifies its use.

FIG. 8 shows the reconstruction of a fluorescent image following thescheme of alternatively capturing the fluorescence, filtering theout-of-focus light and shifting the illumination pattern to a newposition according to the layout in FIG. 5. The quality of the finalreconstructed image is comparable to that produced by traditionalconfocal microscopes based on physical pinholes. The reconstruction ofthe final image as a sum of digitally pinholed emission frames. Theexcitation spot array is shifted to 16×16=256 locations that jointly,completely scan the sample.

FIG. 9 illustrates an issue that arises when several holograms need tobe displayed in a sequence (for instance, in a separable decomposition).Firstly, holograms need to be repeated by the AWG 13 a number of cyclesin a row before switching to the next hologram in the series, in orderfor them to modulate the light beam efficiently. In effect, if thetravelling holograms are displayed just once, they are only correctlyreconstructed at a single point in time: when the hologram fullytraverses the optical window. Before that point, it is only partiallydisplayed as it is still travelling through the exposed laser area.After that point, the first hologram starts to leave the illuminatedarea and the subsequent hologram starts to be displayed (FIG. 9),partially sharing the AOD window temporarily, which causes crosstalk.The diffraction patterns that are generated during the transitionsbetween consecutive holograms reduce the reconstruction quality.

On the contrary, the reconstruction is optimal while the signal of onehologram is continuously repeated, since the circular shifts produced bythe travelling nature of the hologram (the repetition makes the fractionof the hologram that disappears from one end of the illuminated windowapparently reappear through the other end), do not affect thereconstruction because of Fourier transform properties.

In order to solve this difficulty, each hologram signal may be repeateda finite number of instances to ensure a reasonable reconstruction time.The ratio between the total time that a single hologram is displayed andthe transition time between two holograms determines the reconstructionquality. As an additional measure one can introduce a blank signalperiod separating each pair of hologram repetitions in order to totallysuppress the transient mixing. However, the reconstruction time isincreased when more repetitions and blanking periods are used, so atrade-off between frame rate and quality has to be made corresponding tothe need of the application.

An additional issue with hologram sequences is related to an incorrectmultiplication of the X and Y holograms. Note that both the X and Ywindows will be displaying the two consecutive holograms during thehologram transitions (X-hologram 1 and X-hologram 2 on AODx 8, andY-hologram 1 and Y-hologram 2 in AODy 9), wrong products (betweenX-hologram 1 and Y-hologram 2, and between X-hologram 2 and Y-hologram1) will be formed. If the X and Y sequences are synchronized this effectis minimized.

However, depending on several factors, such as a differential soundspeed in the two AO crystals or the spatial centring of the illuminatedareas in the two AOD devices, the RF signals (which are injectedsimultaneously into the two devices) will produce travelling hologramsthat will reach the laser beam at two different times.

This has the effect to temporarily extend the transient period in whichthe hologram in the X-axis multiplies the wrong hologram in the Y-axis(the first X hologram in the sequence multiplies the second Y hologramin the sequence, for example). Therefore, failing to align the hologramsinduce artifacts in the final reconstructed image.

Thus an important feature of the AWG 13 may be incorporation ofadvancing or delaying the hologram sequence in one channel relative tothe other (a relative phase delay control) in order to compensate forthese acoustic path differences in the AOD cells so that one canaccurately align the signal of AODx 8 to that of AODy 9, as illustratedin FIG. 10.

Triangular spot patterns are not separable. However, one can computeseparable holograms that when repeated indefinitely produce the spotpatterns in FIG. 10b and FIG. 10 c.

When these two same holograms are displayed in a sequence during theintegration time of the camera, considering repetitions and blankingperiods as discussed above, and with the appropriate phase delay betweenthe two AWG channels, FIG. 10a shows a non-separable triangular spotarray correctly formed. By contrast, FIG. 10d and FIG. 10e showincorrect reconstructions of the triangular spot array with settings forthe phase delay that do not pre-compensate for the actual acoustic pathdifferences in the setup.

Although only a number of examples have been disclosed herein, otheralternatives, modifications, uses and/or equivalents thereof arepossible. Furthermore, all possible combinations of the describedexamples are also covered. Thus, the scope of the present disclosureshould not be limited by particular examples, but should be determinedonly by a fair reading of the claims that follow. If reference signsrelated to drawings are placed in parentheses in a claim, they aresolely for attempting to increase the intelligibility of the claim, andshall not be construed as limiting the scope of the claim.

1. A programmable multiple-point illuminator for an optical microscope,comprising a light source and a spatial light modulator to modulate alight beam from the light source, the modulated light beam disposed toscan across a sample placed under an objective of the microscope, thesample being provided with fluorophores, the spatial light modulatorcomprising a first acousto-optic deflector and a second acousto-opticdeflector, the first acousto-optic deflector having a first modulationplane and the second acousto-optic deflector having a second modulationplane, said two acousto-optic deflectors being arranged in cascade toprovide respective deflection in different directions, the spatial lightmodulator being enabled to scan in two dimensions across the sample; thespatial light modulator further comprising a telescope relay toconjugate the first modulation plane with the second modulation plane;the illuminator also comprising an arbitrary waveform generator disposedto synthesize holograms and arranged to simultaneously inject a firstsuch hologram into the first acousto-optic deflector and a second suchhologram into the second acousto-optic deflector, for the spatial lightmodulator to modulate the light beam in response to said holograms. 2.The illuminator according to claim 1, the arbitrary waveform generatorbeing a multi-channel generator, at least one channel being arranged toinject a hologram into the first acousto-optic deflector and at leastanother channel being arranged to inject a hologram into the secondacousto-optic deflector, and the arbitrary waveform generator beingdisposed to advance or delay a hologram sequence in one channel withrespect to a hologram sequence in the other channel, to compensate foracoustic path differences in the acousto-optic cells and align thehologram to the first acousto-optic deflector with the hologram to thesecond acousto-optic deflector.
 3. The illuminator according to claim 1,the arbitrary waveform generator being configured to synthesizeholograms of an arbitrary complexity.
 4. The illuminator according toclaim 1, further comprising a tip-tilt opto-mechanical mount for eachacousto-optic deflector.
 5. The illuminator according to claim 1, thelight source comprising a pulsed laser, the illuminator furthercomprising a dispersion compensator to avoid or reduce large groupvelocity dispersion inside the acousto-optic deflectors.
 6. Theilluminator according to claim 1, further comprising a scanning lensarranged after the spatial light modulator to project a reconstructionof a desired illumination pattern onto an intermediate image plane thescanning lens forming, together with a tube lens of the microscope, a 4foptical system to conjugate the modulation planes of the acousto-opticdeflectors with the input pupil of the microscope objective.
 7. Aconfocal filter for an optical microscope having the illuminator ofclaim 2, comprising an imaging sensor provided with an electronicmulti-pixel detector configured to enable real-time implementation ofone digital pinhole around the image of any excited fluorescencelocation in the sample, further comprising a relay system to focus thefluorescent light emitted by the sample on the imaging sensor.
 8. Aconfocal microscope comprising the confocal filter of claim 7,comprising a synchronizer to synchronize the arbitrary waveformgenerator with the imaging sensor in order to correctly compose aconfocal emission image.
 9. A method of operating the confocalmicroscope of claim 8, comprising: making the light source emit a firstlight beam of a certain diameter (D₁); expanding the first light beaminto a second light beam having a prescribed diameter (D₂) to define anillumination window on the first acousto-optic deflector; injecting afirst hologram into the first acousto-optic deflector to modulate thesecond light beam and transform it into a third light beam; imaging thethird light beam on the second modulation plane; collimating the thirdlight beam at zero modulation, so that the diameter (D₂) of the thirdlight beam is that of the second light beam and defines an illuminationwindow on the second acousto-optic deflector; injecting a secondhologram into the second acousto-optic deflector, to modulate the thirdlight beam and transform it into a fourth light beam; imaging the fourthlight beam on the input pupil of the microscope objective; focusing thefourth light beam on a reconstruction plane that intersects the sample;and collecting the fluorescent light emitted by the sample and focusingsaid light on the imaging sensor.
 10. The method according to claim 9,further comprising: computing the first synthetic radiofrequency signalwith a first digital holography algorithm and the second syntheticradiofrequency signal with a second digital holography algorithm; andsynthetizing, by the arbitrary waveform generator, the first hologram,to be injected into the first acousto-optic deflector, from said firstcomputed signal, and the second hologram, to be injected into the secondacousto-optic deflector, from said second computed signal.
 11. Themethod according to claim 10, further comprising advancing or delayingthe hologram sequence in a channel of the arbitrary waveform generatorsynthetizes the first or second hologram, with respect to the hologramsequence in another channel of the arbitrary waveform generator thatsynthetizes the other hologram.
 12. The method according to claim 10,further comprising repeating each hologram a number of cycles in a rowand introducing a blank period before switching to another hologram. 13.The method according to claim 10, further comprising decomposing thehologram into separable components.
 14. The method according to claim 9,further comprising synchronizing the arbitrary waveform generator withthe imaging sensor to correctly compose a confocal image.
 15. The methodaccording to claim 13, further comprising composing the confocal imageas a sum of several processed emission images excited by mathematicallyseparable illumination patterns.
 16. The illuminator according to claim2, the arbitrary waveform generator being configured to synthesizeholograms of an arbitrary complexity.
 17. The illuminator according toclaim 2, further comprising a scanning lens arranged after the spatiallight modulator to project a reconstruction of a desired illuminationpattern onto an intermediate image plane, the scanning lens forming,together with a tube lens of the microscope, a 4f optical system toconjugate the modulation planes of the acousto-optic deflectors with theinput pupil of the microscope objective.
 18. The confocal filteraccording to claim 7, the arbitrary waveform generator being amulti-channel generator, at least one channel being arranged to inject ahologram into the first acousto-optic deflector and at least anotherchannel being arranged to inject a hologram into the secondacousto-optic deflector, and the arbitrary waveform generator beingdisposed to advance or delay a hologram sequence in one channel withrespect to a hologram sequence in the other channel, to compensate foracoustic path differences in the acousto-optic cells and align thehologram to the first acousto-optic deflector with the hologram to thesecond acousto-optic deflector.
 19. The confocal microscope according toclaim 8, the arbitrary waveform generator being a multi-channelgenerator, at least one channel being arranged to inject a hologram intothe first acousto-optic deflector and at least another channel beingarranged to inject a hologram into the second acousto-optic deflector,and the arbitrary waveform generator being disposed to advance or delaya hologram sequence in one channel with respect to a hologram sequencein the other channel, in order to compensate for acoustic pathdifferences in the acousto-optic cells and align the hologram to thefirst acousto-optic deflector with the hologram to the secondacousto-optic deflector.
 20. The method according to claim 9, thearbitrary waveform generator being a multi-channel generator, the methodfurther comprising: through at least one channel, injecting a holograminto the first acousto-optic deflector; through at least anotherchannel, injecting a hologram into the second acousto-optic deflector;advancing or delaying a hologram sequence in one channel with respect toa hologram sequence in the other channel, in order to compensate foracoustic path differences in the acousto-optic cells; aligning thehologram into the first acousto-optic deflector with the hologram intothe second acousto-optic deflector.